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Summary 


The  Defense  Advanced  Research  Projects  Agency  (DARPA)  Education  Dominance  Pro¬ 
gram  is  a  research  effort  intended  to  substantially  improve  Defense  training  and  human  perfor¬ 
mance  by  capturing  in  scalable  computer  technology  the  best  practices  of  human  tutors.  Among 
other  benefits,  this  approach  promises  to  significantly  compress  the  time  needed  by  technicians 
to  achieve  high  levels  of  expertise.  This  effort  focused  on  the  Navy’s  Information  Systems  Tech¬ 
nician  (IT)  rating.  DARPA,  through  its  research  contractor,  undertook  the  development  of  a  Dig¬ 
ital  Tutor  (DT)  that  would  capture  in  computer  technology  the  best  instructional  practices  of 
highly  capable  tutors. 

DT  development  was  planned  for  two  phases.  Phase  1  ended  on  30  September  2009. 
Initiation  of  Phase  2  depended  on  findings  from  Phase  I .  During  Phase  I ,  representative  “A” 
School  IT  candidates  were  trained  in  cooperation  with  the  Navy’s  Center  for  Information 
Dominance  (CID)  in  a  16-week  Advanced  Concepts  in  Education  (ACE)  program  of  instruction. 
About  2  weeks  of  instruction  was  presented  by  an  early  available  component  of  the  (computer¬ 
ized)  DT.  The  remaining  14  weeks  of  the  course  were  presented  by  highly  qualified  human 
tutors.  ACE/CID  training  focused  on  core  networking  competencies  derived  from  IT  “A”  School 
content,  substantial  portions  of  five  relevant  “C”  schools,  and  a  careful  analysis  of  Elect  IT 
requirements. 

The  challenge  for  this  training  was  to  show  that  its  students  were  at  least  equivalent  in 
knowledge  and  skill  to  Elect  ITs  with  4-12  years  experience.  If  this  challenge  was  met,  develop¬ 
ment  of  the  fully  computerized  DT  (Phase  2)  was  scheduled  to  begin  1  October  2009  and  end 
30  September  2011.  Phase  1  Information  Warfare  (IWAR)  testing  was  conducted  to  assess  this 
possibility  and  consisted  of  three  activities: 

1.  IWAR-Laboratory  (IWAR-L),  This  activity  compared  the  technical  capabilities  of 
ACE/CID  students  with  experienced  Elect  ITs  in  highly  instrumented  laboratory 
conditions  supplemented  by  human  observers 

2.  IWAR-Dockside  (IWAR-D).  This  activity  assessed  the  technical  capabilities  of 
ACE/CID  students  aboard  docked  Navy  ships 

3.  IWAR-Afloat  (IWAR-A).  This  activity  assessed  the  technical  capabilities  of  ACE/ 
CID  students  aboard  deployed  Navy  ships. 
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ACE/CID  training  of  12  “A”  School  candidates  was  completed  on  21  May  2009.  No  dif¬ 
ferences  of  statistical  or  practical  significance  in  age  or  Armed  Forces  Qualification  Test 
(AFQT)  scores  were  found  between  the  ACE/CID  students  and  typieal  “A”  Sehool  students.  CID 
prepared  and  condueted  a  test  to  ensure  that  ACE/CID  students  were  receiving  adequate  IT  “A” 
School  instruction.  Scores  of  the  ACE/CID  students  on  the  test  were  significantly  higher  (p  < 
.01)  than  those  of  a  cohort  of  “A”  School  students. 

A.  IWAR-L 

Twelve  ITs  were  detailed  from  the  Fleet  to  participate  in  IWAR-E  testing.  They  averaged 
8.5  years  of  Navy  experienee  and  7.2  years  of  Navy  IT  experience.  Overall,  they  had  completed 
41  “C”  schools.  Their  AFQT  scores  were  slightly  higher  than,  but  effectively  equivalent  to,  those 
of  the  ACE/CID  students. 

IWAR-E  was  condueted  in  two  5-day  cycles:  13-17  July  2009  and  27-31  July  2009.  Six 
ACE/CID  students  and  six  Fleet  ITs  participated  in  Cycle  1,  and  the  remaining  six  ACE/CID 
students  and  six  Fleet  ITs  participated  in  Cycle  2.  CID  made  three  classrooms  available  for 
IWAR-E  at  the  San  Diego  Naval  Base.  Two  classrooms  were  used  for  testing,  and  one  classroom 
housed  the  hardware  and  software  needed  to  manage  and  operate  systems  in  the  testing  rooms. 
Systems  in  the  testing  rooms  were  made  to  resemble  shipboard  Navy  systems  as  closely  as 
possible. 

IWAR-E  was  divided  into  four  major  activities: 

1.  Knowledge  Testing  (4  hours,  Day  1).  Knowledge  Testing  consisted  almost  entirely 
of  short-answer  questions  on  12  topics.  The  ACE/CID  students  received  higher 
seores  on  11  of  the  12  test  topics.  Nine  of  these  differenees  were  statistically  signifi¬ 
cant  (p  <  .05  and  p  <  .01)  as  were  the  total  scores  (p  <  .01)  reeeived  by  the  ACE/CID 
students. 

2.  Troubleshooting  (13.25  hours,  Days  1-3).  Almost  half  of  IWAR-E  was  conducted 
as  Troubleshooting,  whieh  was  viewed  as  the  most  realistie,  representative,  and  valid 
assessment  of  IT  readiness  for  Fleet  duty.  Most  IWAR-E  troubleshooting  problems 
were  based  on  an  analysis  of  1,093  Casualty  Report  (CASREP)  trouble  tickets  drawn 
from  mueh  larger  Fleet  Remedy  databases. 

The  six  Fleet  ITs  and  six  ACE/CID  students  in  each  cycle  were  divided  into  two 
three-member  teams,  one  of  whieh  worked  on  virtual  problems  and  one  of  which 
worked  on  physieal  problems.  Fleet  ITs  and  the  ACE/CID  students  were  assigned 
the  same  troubleshooting  problems  in  the  same  sequence. 

Overall,  Fleet  ITs  attempted  95  troubleshooting  problems  and  successfully  solved  79 
(83%)  of  them,  with  46  (58%)  of  their  solutions  rated  exeellent.  ACE/CID  students 
attempted  102  troubleshooting  problems  and  suecessfully  solved  99  (97%)  of  them. 
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with  89  (90%)  of  their  solutions  rated  exeellent.  The  Fleet  ITs  made  18  harmful 
changes  in  the  process  of  troubleshooting,  and  the  ACE/CID  students  made  8  harm¬ 
ful  changes  in  the  process  of  troubleshooting.  The  ACE/CID  students  were  more 
systematic  in  their  troubleshooting  processes  as  represented  by  97%  problems  and 
95%  solutions  verified  compared  to  85%  problems  and  77%  solutions  for  the  Elect 
ITs. 

3.  Security  Testing  (7  hours,  Day  4)T  Eor  this  exercise,  the  physical  systems  in  each 
testing  room  were  pre-loaded  with  about  100  typical  shipboard  security  vulnerabili¬ 
ties.  Elect  and  ACE/CID  teams  were  to  restore  the  systems  to  Elect  security  specifi¬ 
cations  while  avoiding  changes  that  did  not  conform  to  Elect  regulations.  No  team 
did  well  on  this  portion  of  IWAR-E.  Elect  teams  scored  149  (35%)  of  the  total  points 
possible,  and  ACE/CID  teams  scored  95  (23%)  of  the  total  points  possible.  These 
differences  do  not  differ  significantly  (p  <  .05). 

4.  Design  and  Implementation  (7  hours,  Day  5). 2  Eor  this  exercise.  Elect  ITs  and 
ACE/CID  students  were  to  assemble  a  system  that  met  24  objectives.  Overall,  Elect 
ITs  achieved  15  of  these  objectives,  and  ACE/CID  students  achieved  14  of  these 
objectives.  Neither  group  did  well  in  scoring.  Elect  ITs  scored  113.5  (47%)  points 
out  of  240  possible,  and  ACE/CID  students  scored  84.5  (35%)  points  out  of 
240  possible.  These  differences  were  not  statistically  significant  (p  <  .05). 

B.  IWAR-D 

IWAR-D  consisted  of  two  5-day  cycles,  3-7  August  and  10-14  August  2009.  Two 
DDGs  were  made  available  for  both  cycles:  the  USS  Preble  (DDG  88)  and  the  USS  Milius 
(DDG  69).  Three  ACE/CID  students  were  assigned  to  each  ship  in  each  cycle.  Data  consisted  of 
student  task  diaries  and  surveys  completed  by  supervisors. 

Overall  the  ACE/CID  students  completed  about  38  tasks  during  IWAR-D.  The  number  of 
tasks  was  affected  by  shipboard  schedules  and  activities.  Most  tasks  assigned  to  the  ACE/CID 
students  were  either  Trouble  Ticket  responses  or  Emerging  Problem  tasks,  which  were  those  that 
were  found  while  correcting  other  problems.  The  ACE/CID  students  solved  about  87%  of  tasks 
assigned  to  them  successfully.  Some  tasks  were  not  solved  because  the  ACE/CID  students  were 
not  permitted  access  to  a  component  or  because  the  time  allocated  for  the  day’s  IWAR-D  testing 
ran  out. 


Most  of  the  contributions  of  the  ACE/CID  students  were  rated  as  substantial  or  essential. 
Two  instances  occurred  in  which  the  ACE/CID  students  solved  problems  that  shipboard  ITs  had 


1  In  the  morning,  half  hour  of  orientation  followed  by  3.5  hours  of  the  exercise  in  the  morning  for  two  6-member 
Fleet  teams  working  in  the  separate  laboratory  rooms.  In  the  afternoon,  a  half  hour  of  orientation  followed  by 
3.5  hours  of  the  exercise  for  two  6-member  ACE/CID  teams  working  in  the  separate  laboratory  rooms. 

2  3.5  hours  morning  testing  and  3.5  hours  afternoon  testing. 
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not  been  able  to  correct  either  with  their  own  solutions  or  with  solutions  they  had  received  from 
the  Space  and  Naval  Warfare  Systems  Center  (SPAWAR),  San  Diego. 

C.  IWAR-A 

IWAR-A  was  limited  to  observations  of  three  ACE/CID  students  who  served  aboard  the 
USS  Vandegrift  (FFG-48),  17-20  August  2009.  IWAR-A  data  consisted  of  student  task  diaries 
and  a  narrative  report  from  the  Senior  Chief  who  served  as  their  monitor. 

The  ACE/CID  students  were  assigned  system-wide  tasks  that  are  typically  addressed  by 
senior  ITs.  These  tasks  were  essential  for  successful  operation  of  the  ship’s  IT  systems  and 
would  have  created  debilitating  results  if  done  incorrectly.  All  were  reported  to  have  been 
accomplished  correctly.  All  three  ACE/CID  students  participated  equally  in  technical  activities. 

D.  Discussion 

The  ACE/CID  training  challenge  appears  to  have  been  met.  On  this  basis,  Phase  2  of  the 
DARPA  program  was  initiated.  If  successful  it  should  advance  the  state  of  training  art  and  prac¬ 
tice  and  contribute  significantly  to  Navy  operational  capabilities. 
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Phase  1 IWAR  Test  Results 


A,  Background 

The  benefits  of  tutorial — one  tutor  for  (ideally)  one  student — instruetion  have  long  been 
known,  understood,  and  doeumented  (e.g..  Bloom,  1984;  Graesser  &  Person,  1994;  Graesser, 
D’Mello,  &  Cade,  in  press).  Tutorial  instruetion  provides  a  variety  of  demonstrated  benefits 
(e.g.,  interaetivity,  individualization,  and  engagement)  and  has  also  been  suggested  as  a  way  to 
reliably  eompress  the  many  years  needed  to  develop  teehnieal  expertise  into  substantially  shorter 
periods  of  time  (Fletcher,  2001,  2009).  However,  with  the  exception  of  a  few  critical  skills,  it  has 
been  viewed  as  an  economic  impossibility  (Scriven,  1975). 

Affordable  computer  technology  and  adaptive  computer-based  instruction  promise, 
through  their  capabilities  for  massive  scalability,  to  make  tutorial  instruction  affordable  (Fletcher 
&  Chatham,  2010).  Research  and  development  (R&D)  that  attempts  to  realize  this  promise  has 
been  supported  since  the  late  1960s,  much  of  it  sponsored  by  the  U.S.  Navy  (Coulson,  1962; 
Fletcher  &  Rockway,  1986;  Ford,  Slough,  &  Hurlock,  1972). 

B,  Defense  Advanced  Research  Projects  Agency  (DARPA)  Education  Dominance 
Program 

Most  research  on  computer-based  tutoring  has  attempted  to  apply  theories  and 
approaches  developed  by  cognitive  and  computer  scientists  (e.g.,  Sleeman  &  Brown,  1982; 
Psotka,  Massey,  &  Mutter,  1988;  Luckin,  Koedinger,  &  Greer,  2007).  The  DARPA  Education 
Dominance  Program  has  taken  another,  more  pragmatic  approach.  Noting  that  large  differences 
can  be  observed  and  measured  in  the  learning  results  achieved  by  different  teachers,  DARPA, 
through  its  research  contractor,  has  chosen  to  examine  and  capture  in  computer  technology  the 
best  practices  of  expert  human  tutors.  The  aim  is  to  create  a  Digital  Tutor  (DT)  that  achieves  or 
even  exceeds  the  “2-Sigma”  challenge  laid  down  by  Bloom  (1984) — a  challenge  to  produce  two 
standard  deviations  of  difference  in  learning  over  more  conventional  approaches.  DARPA  is 
focusing  this  effort  on  training  that  is  intended  to  prepare  sailors  for  the  Information  Systems 
Technician  (IT)  rating.  Roughly,  a  two  standard  deviation  difference  would  raise  the  perfor¬ 
mance  of  IT  technicians  at  the  50th  percentile  to  that  of  the  98th  percentile. 

DT  development  was  planned  for  two  phases.  Phase  1  ended  30  September  2009.  During 
Phase  1,  representative  “A”  school  IT  candidates  were  to  be  trained  using  DT  content  and 
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methodology  in  a  16-week  Advanced  Concepts  in  Education  (ACE)  program.  This  training  was 
completed  in  May  2009. 

C.  Phase  1  Information  Warfare  (IWAR) 

IWAR  testing  was  to  determine  if  Phase  1  training  had  produced  IT  technicians  who  have 
the  technical  knowledge  and  skills  of  technicians  who  have  4-12  years  experience  in  the  Elect.  If 
IWAR  showed  that  the  DARPA  approach  had  met  this  objective,  Phase  2  would  begin  develop¬ 
ment  and  testing  of  the  fully  computerized  DT  on  1  October  2009.  Similar  IWAR  testing  would 
be  conducted  at  the  conclusion  of  Phase  2. 

As  indicated  by  the  data  presented  in  this  report,  Phase  1  successfully  met  and,  in  fact, 
exceeded  its  objectives.  Phase  2  of  the  DARPA  program  was  initiated  on  1  October  2009. 

The  training  and  testing  required  for  this  effort  were  made  possible  through  cooperation 
with  the  Center  for  Information  Dominance  (CID)  Corry  Station,  Pensacola,  Elorida.  CID  sup¬ 
plied  (1)  students  from  the  “A”  school  pool  at  Corry  Station,  (2)  space  and  facilities  to  train  them 
with  DT  content  and  techniques  at  the  CID  Detachment,  Monterey,  California,  and  (3)  space  and 
facilities  for  IWAR  assessment  at  the  CID  detachment  at  the  San  Diego  Naval  Base.  The  Navy 
IT  candidates  who  participated  in  this  research  are  designated  as  ACE/CID  students  in  this 
report. 

Three  assessment  activities  were  undertaken  for  Phase  1  IWAR  testing: 

1.  IWAR-Laboratory  (IWAR-L).  Knowledge  and  performance  assessment  of  the 
ACE/CID  students  compared  with  experienced  Elect  ITs.  Knowledge  assessment 
was  accomplished  using  written,  short-answer  response  paper  and  pencil  tests.  Per¬ 
formance  testing  was  conducted  with  both  groups  operating,  troubleshooting, 
securing,  designing,  and  assembling  Elect  systems  and  equipment  in  highly  instru¬ 
mented  laboratory  conditions  supplemented  by  human  observers. 

2.  IWAR-Dockside  (IWAR-D).  Dockside  performance  of  the  ACE/CID  students  with 
shipboard  systems  and  human  observers,  as  permitted  by  shipboard  conditions. 

3.  IWAR- At  Sea  (IWAR- A),  At-sea  performance  of  the  ACE/CID  students  with  ship¬ 
board  systems  and  human  observers,  as  permitted  by  deployment  and  shipboard 
conditions. 

This  report  describes  these  three  activities  and  summarizes  the  findings.  The  testing  pro¬ 
vided  a  controlled  and  complete  comparison  between  ACE/CID  students  and  Elect  ITs.  IWAR-L 
and  IWAR-D  testing  took  place  in  San  Diego  from  July-September  2009.  IWAR-A  took  place  at 
sea  during  September  2009. 
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D.  ACE/CID  Training 

ACE/CID  training  began  in  January  2009.  Most  of  the  training  was  presented  by  about 
two  dozen  human  tutors  who  possessed  expertise  in  one  or  more  IT  subdomains  eovered  by  the 
training.  Half  of  these  tutors  were  seleeted  through  extensive  interviews  that  sought  to  aseertain 
instructional  expertise  (demonstrated  in  half-hour  tutoring  “auditions”)  and  subject-matter  mas¬ 
tery  of  an  IT  subdomain  needed  for  the  ACE/CID  training.  Another  quarter  of  the  instructors  had 
written  a  definitive,  advanced  text  for  an  appropriate  IT  subdomain  and  were  then  selected  for 
their  tutorial  expertise.  The  final  quarter  of  instructors  were  selected  on  the  recommendation  of 
qualified  organizations  that  identified  them  for  their  tutorial  and  IT  subdomain  expertise. 

Per  plan,  an  early  available  portion  of  the  (computerized)  DT  presented  1-2  weeks  of  the 
ACE/CID  training.  Time  of  use  depended  on  that  needed  by  individual  students  to  master  the 
material.  The  goal  was  for  the  ACE/CID  students  to  receive  1:1  tutoring  from  the  human  tutors 
for  the  remaining  instruction.  Major  exceptions  were  a  1-week  hardware  laboratory,  which  was 
taught  1:4;  a  half  week  of  networking,  which  was  taught  1:8  because  of  limited  tutor  availability; 
and  9  students  who  received  1 :2  tutoring  for  about  half  of  their  instruction,  again  due  to  limited 
availability  of  the  (human)  tutor. 

About  30%  of  ACE/CID  instructional  content  was  based  on  a  careful  analysis  of  Elect  IT 
requirements  undertaken  in  2008.  The  remaining  two-thirds  of  the  instructional  content  was 
derived  from  topics  covered  by  the  1 1 -week  CID  IT  “A”  school  and  the  core  material  covered  in 
the  following  5  “C”  schools: 

1.  Journeyman-Networking  Core  (JNETCORE) 

2.  Advanced  Network  Analyst  (ANA) 

3.  Information  Systems  Security  Manager  (ISSM) 

4.  Network  Security  Vulnerability  Technician  (NSVT) 

5.  Navy  Tactical  Command  Support  System  (NTCSS)  Manager. 

ACE/CID  objectives  focused  on  developing  core  competencies  in  networking  (i.e.,  work¬ 
station  and  server  software,  routing  and  switching,  operating  systems,  and  so  forth).  This  training 
was  also  designed  to  provide  students  the  knowledge,  understanding,  and  general  problem¬ 
solving  abilities  that  would  enable  them  to  learn  and  adapt  to  new  systems  and  configurations 
quickly. 
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ACE/CID  training  covered  the  following  basic  topics: 

•  Hardware:  Windows  Basics 

•  Windows  2003  Server:  Primary  Services 

•  Windows  2003  Server:  Applieations 

•  Fleet  Networking,  Routing,  and  Switching 

•  Switching  (CISCO/ Alcatel) 

•  Radio  Frequeney  (RF)  Communications 

•  Network  Security 

•  Programming. 

ACE/CID  training  took  place  from  26  January  to  21  May  2009.  Three  days  in  week  17 
were  added  to  make  up  for  holidays  during  the  16-week  period.  Twelve  students  completed  the 
ACE/CID  training.  No  students  were  dropped  from  the  program  beeause  of  aeademie  reasons. 

E.  IWAR-L 

1.  IWAR-L  Organization 

IWAR-F  provided  the  first  and  most  extensive  testing  for  Phase  I  of  the  DARPA  pro¬ 
gram.  It  eompared  the  knowledge  and  skills  of  the  12  ACE/CID  students  who  had  completed 
ACE/CID  training  with  the  knowledge  and  skills  of  12  Fleet  ITs  who,  as  it  turned  out,  had 
3-12  years  of  IT  experienee.  The  Fleet  ITs  were  detailed  from  large-deck  and  small-deck  ships. 

IWAR-E  was  undertaken  and  eompleted  in  two  5-day  cycles:  13-17  July  and  27-31  July 
2009.  Six  ACE/CID  students  and  six  Fleet  ITs  partieipated  in  Cycle  1.  The  remaining  six 
ACE/CID  students  and  six  Fleet  ITs  participated  in  Cycle  2. 

2,  IWAR-L  Support  Teams 

Three  teams  supported  IWAR-L  testing: 

•  White  Team,  The  White  Team  eonducted  the  IWAR-L  exereises.  White  Team 
members  organized  partieipants  for  IWAR  trials,  ensured  that  they  observed  trial 
parameters  and  protocols,  managed  information  eommunication,  and  participated  in 
recording  and  scoring  performance  data.  IWAR-L  was  espeeially  fortunate  in 
obtaining  the  serviees  of  the  White  Team  leader,  two  CID  instructors,  and  three  Fleet 
Systems  Engineering  Team  (FSET)  members  for  this  work. 

•  Red  Team,  The  Red  Team  consisted  of  a  highly  experieneed  Senior  Chief,  who  was 
assisted  by  equally  capable  FSETs.  The  Red  Team  designed  and  set  up  systems  for 
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IWAR-L  security  testing  and  helped  assess  the  performance  of  participating  ITs  in 
correcting  security  vulnerabilities. 

•  Technical  Team,  The  Technical  Team  was  provided  by  the  research  contractor  and 
was  responsible  for  the  proper  management  and  operation  of  the  IWAR-L  hardware 
and  software.  Commendably,  especially  for  such  a  complex  integration  of  many  dif¬ 
ferent  systems,  IWAR-L  testing  did  not  experience  any  major  technical  disruptions 
during  its  two  5-day  cycles  of  exercises. 

3.  IWAR-L  Testing  Laboratories 

IWAR-L  participants  worked  in  two  large  separate  laboratory  rooms  (each  approximately 
20  ft.  X  30  ft.)  provided  by  CID  on  the  San  Diego  Naval  Base.  Each  laboratory  room  contained 
three  Information  Technology  systems:  two  identical  virtual  systems  that  participating  ITs  oper¬ 
ated  via  keyboard  and  monitor  displays  and  one  physical  system  with  cables  and  other  hardware 
that  participants  operated  manually.  The  laboratory  systems  were  designed  to  mirror  typical 
shipboard  Information  Technology  equipment  (servers,  routers,  switches,  and  so  forth).  A  third 
classroom  housed  the  hardware  and  software  that  the  Technical  Team  needed  to  manage  and 
operate  systems  in  the  testing  rooms. 

The  two  virtual  systems  consisted  of  real  server  and  networking  software  (e.g., 
COMPOSE  3.0  running  on  three  servers  with  a  backbone)  but  without  the  capacity  of  a  full  hard¬ 
ware  implementation.  Three  workstations  were  used  to  operate  each  of  the  virtual  systems.  The 
physical  laboratory  system  had  a  server  rack  with  four  servers,  one  UNIX  system,  two  backbone 
switches,  and  four  edge  switches.  Connected  to  the  rack  were  three  workstations  on  a  network 
made  to  resemble  shipboard  systems  as  closely  as  possible. 

The  Technical  Team  was  responsible  for  injecting  troubleshooting  problems  and  cor¬ 
recting  any  system  technical  issues  that  arose.  The  Technical  Team  ensured  that  the  systems 
could  be  restarted  quickly — when  and  if  that  was  required.  Spare  hardware  for  every  system 
component  was  available,  along  with  spare  disks  with  cloned  images  for  the  servers  so  that 
problems  with  the  server  rack  would  only  take  a  few  minutes  to  resolve. 

The  laboratories  were  also  instrumented  with  video  cameras  and  microphones.  All  activ¬ 
ity  on  the  monitors  was  time  stamped  for  later  review  and  analyses  using  tools  already  developed 
for  ACE/CID  training. 

4,  IWAR-L  ACE/CID  Student  and  Fleet  IT  Teams 

Eor  much  of  IWAR-L  testing,  the  six  participants  in  the  two  laboratory  rooms  were 
divided  into  two  three-member  teams.  Simultaneously,  one  team  worked  on  virtual  systems  to 
solve  system  problems,  and  the  other  team  worked  on  physical  problems.  Members  of  these 
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teams  were  assigned  by  the  White  Team.  For  the  Fleet  teams,  one  member  was  an  ITl  with 
11+  years  of  experienee,  a  seeond  member  was  an  IT2  with  6+  years  of  experienee,  and  the  third 
member  was  an  IT2  with  less  than  6  years’  experienee.  For  the  ACE/CID  teams,  one  member 
was  assigned  from  the  top  third  of  the  ACE/CID  students,  one  member  from  the  middle  third, 
and  one  member  from  the  lower  third. 

5.  IWAR-L  Schedule 

Table  1  shows  the  daily  sehedule  followed  for  eaeh  of  the  two  5 -day  IWAR-E  eyeles. 


Table  1.  IWAR-L  Daily  Schedule 


Day  1 

Day  2 

Day  3 

Day  4 

Day  5 

•  Introduction  to 

IWAR  testing 
(0.5  hour) 

•  Knowledge  Test 
Part  1  (2  hours) 

Break  (30  min) 

•  Knowledge  Test 
Part  2  (2  hours) 

Lunch 

•  Familiarization 
with  laboratory 
systems  (1  hour) 

•  Troubleshooting 
very  easy  to 
medium  difficulty 
problems 

(2  hours) 

•  Troubleshooting 
easy,  medium, 
and  difficult  prob¬ 
lems  (6  hours) 

•  Troubleshooting 
medium  to  very 
difficult  problems 
(5.25  hours) 

•  Security:  Identifi¬ 
cation  and  correc¬ 
tion  of  security 
vulnerabilities 

•  Morning  orienta¬ 
tion  (0.5  hour) 

•  Morning  Fleet 
team  testing 
(3.5  hours) 

Lunch 

•  Afternoon  orienta¬ 
tion  (0.5  hour) 

•  Afternoon 

ACE/CID  team 
testing  (3.5  hours) 

•  Design  and  imple¬ 
mentation:  Given 
objectives,  design 
and  implement  an 

IT  system 

•  Morning  testing 
(3.5  hours) 

Lunch 

•  Afternoon  testing 
(3.5  hours) 

As  Table  1  shows,  the  IWAR-E  schedule  involved  the  following  activities: 

•  Day  1 

-  Introduction  to  I  WAR  testing  (0.5  hour) 

-  Two-part  written  test  of  IT  knowledge  (4  hours) 

-  Eamiliarization  with  the  laboratory  systems  (1  hour) 

-  Troubleshooting  very  easy  to  medium  difficulty  problems  (2  hours) 

•  Day  2 

-  Troubleshooting  easy,  medium,  and  difficult  problems  (6  hours) 

•  Day  3 

-  Troubleshooting  medium,  difficult,  and  very  difficult  problems  (5.25  hours) 
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•  Day  4 

-  Finding  and  correcting  security  vulnerabilities.  In  the  morning,  a  half  hour  of  orien¬ 
tation  was  followed  by  3.5  hours  of  the  exercise  in  the  morning  for  two  6-member 
Fleet  teams  working  in  the  separate  laboratory  rooms.  In  the  afternoon,  a  half  hour  of 
orientation  was  followed  by  3.5  hours  of  the  exercise  for  two  6-member  ACE/CID 
teams  working  in  the  separate  laboratory  rooms. 

•  Day  5 

-  System  design  and  implementation  (3.5  hours  in  the  morning  and  3.5  hours  in  the 
afternoon).  Six-member  Fleet  and  ACE/CID  teams  working  in  the  separate  labora¬ 
tory  rooms. 

To  ensure  standardization  of  treatment,  a  member  of  the  White  Team  introduced  each 
day’s  session  with  a  written  “script”  that  was  read  to  the  participating  ITs.  The  same  trouble¬ 
shooting  problems  (Days  1-3)  were  given  to  both  groups  in  exactly  the  same  sequence.  Different 
troubleshooting  items  were  presented  in  the  two  cycles,  but  they  followed  the  same  sequence  of 
difficulty  and  impact  in  both  cycles. 

6.  Comparisons  of  ACE/CID  Students  and  “A”  School  Students 

Thirty-two  students  (22  males  and  10  females)  who  had  been  selected  for  IT  “A”  school 
training  were  offered  as  candidates  for  ACE/CID  training.  Eifteen  were  selected  (11  males  and 
4  females).  The  candidate  who  had  the  highest  Armed  Eorces  Qualification  Test  (AEQT)  score 
was  not  selected  for  ACE/CID  training,  and  a  student  who  required  a  waiver  for  admission  to 
“A”  school  was  selected.  Two  of  the  students  chosen  for  ACE/CID  training  had  learned  English 
as  a  second  language.  None  of  the  students  selected  reported  any  IT  experience  before  their  Navy 
enlistment. 

Table  2  provides  the  data  available  on  the  15  students  selected  for  ACE/CID  training  and 
on  the  remaining  “A”  school  students  who  were  not  selected. 


Table  2.  ACE/CID  Students  Compared  With  “A”  School  Students 


Age 

AFQT  Scores 

ACE/CID 

Students 

Remaining 
“A”  School  Students 

ACE/CID 

Students 

Remaining 
“A”  School  Students 

Average 

20.8 

19.6 

71.3 

69.8 

Std  dev 

2.7 

2.2 

11.8 

11.5 

N 

15 

17 

15 

17 
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The  data  on  age  and  AFQT  seores  for  students  seleeted  for  ACE/CID  training  did  not  dif¬ 
fer  statistieally  from  those  who  were  not  seleeted.  Further,  as  shown  in  Table  3,  the  ACE/CID 
students  shown  in  Table  2  did  not  differ  statistieally  nor  did  they  appear  to  differ  praetieally  from 
the  2008  and  2009  populations  of  “A”  School  students. 


Table  3.  Average  Age  and  AFQT  Scores  of  CID  IT  “A”  School  Students 


Age  of 

CID  IT  “A”  School  Students 

AFQT  Scores  of 

CID  IT  “A”  School  Students 

FY2008 

FY2009 

FY2008 

FY2009 

Average 

20.9 

21.1 

72.8 

73.3 

Std  dev 

2.8 

3.2 

12.7 

12.4 

N 

517 

1,172 

517 

1,155 

Note  for  Table  3:  Scores  not  available  for  all  2009  students. 


As  ACE/CID  training  progressed,  CID  command  wanted  to  be  assured  that  its  students 
were  receiving  adequate  levels  of  IT  instruction.  Accordingly,  CID  developed  a  test  containing 
essay,  diagram,  and  multiple-choice  questions  that  were  administered  to  ACE/CID  students  and 
“A”  school  students.  The  test  covered  the  5  weeks  of  network  instruction  received  by  the  “A” 
school  students  and  the  2  weeks  of  similar  material  received  by  the  ACE/CID  students.  Table  4 
shows  the  results  of  this  testing. 


Table  4.  Performance  of  ACE/CID  and  “A”  School  Students  on  CID  IT  Test 


Essay 

Diagram 

Multiple-Choice 

Totals 

ACE/CID 

“A” 

School 

ACE/CID 

“A” 

School 

ACE/CID 

“A” 

School 

ACE/CID 

“A” 

School 

Mean 

49.7 

21.5 

13.3 

6.575 

14.7 

11.6 

77.7 

39.7 

Std  Dev 

8.9 

12.65 

3.2 

3.77 

1.6 

3.5 

11.8 

18.7 

N 

15 

20 

15 

20 

15 

20 

15 

20 

Probability 

p  <  .01 

p  <  .01 

p  <  .01 

p  <  .01 

Effect  Size 

2.5 

1.9 

1.1 

2.36 

All  15  ACE/CID  students  took  the  CID  IT  test  (3  students  were  later  dropped  from  the 
ACE/CID  training  program  for  non-academic  reasons).  The  differences  on  all  the  subtests  and 
the  total  test  scores  are  statistically  significant  (p  <  .01)  in  favor  of  the  ACE/CID  students.  These 
results  suggest  that  the  ACE/CID  students  had  successfully  learned  standard  “A”  school  net¬ 
working  content  during  their  first  2  weeks  of  training  and  before  moving  on  to  content  drawn 
from  “C”  schools  and  Fleet  requirements.  Their  average  score  was  almost  twice  that  of  the  “A” 
school  students. 
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Effect  sizes  are  reported  in  Table  4  and  elsewhere  in  this  report.  Increasingly,  effect  sizes 
are  being  required  and  reported  for  single  research  studies.  These  statistics,  as  their  name  sug¬ 
gests,  are  used  to  go  beyond  the  point  comparisons  of  statistical  significance  and  express  the 
magnitude  of  treatment  or  experimental  effect  along  a  standardized  continuum. 

An  effect  size  is  usually  calculated  as  the  difference  between  two  means  divided  by  their 
pooled  standard  deviation.  It  therefore  expresses  treatment  effect  magnitude  as  a  standard  devia¬ 
tion.  An  effect  size  of  0.20  standard  deviations  is  considered  small.  Roughly,  it  may  be  viewed  as 
improving  performance  from  the  50th  to  about  the  58th  percentile  of  performance.  An  effect  size 
of  0.50  is  generally  considered  to  be  medium  and  can  be  viewed  as  an  improvement  from  the 
50th  to  about  the  69th  percentile  of  performance.  An  effect  size  of  0.80  is  considered  large  and 
can  be  viewed  as  an  improvement  to  about  the  79th  percentile  of  performance  (Cohen,  1988). 

One  challenge  for  the  DARPA  Education  Dominance  Program  was  expressed  in  these 
units.  Based  on  Bloom’s  (1984)  often  cited  challenge,  it  assumed  a  goal  of  making  a  two  stan¬ 
dard  deviation  improvement  over  current  “A”  school  IT  education  and  training.  This  goal  is 
ambitious,  although  it  is  not  a  ceiling.  Research  may  aspire  to  whatever  effect  size  seems 
feasible. 

Notably,  the  effect  size  of  2.36  on  the  CID  test  in  favor  of  the  ACE/CID  students  (see 
Table  4)  well  exceeds  the  two  standard  deviation  goal.  Given  the  results  reported  from  the 
Knowledge  Test  (see  Table  6),  which  compared  performance  of  ACE/CID  students  with  Elect 
ITs,  it  seems  likely  that  if  “A”  School  students  had  taken  the  test  instead,  the  effect  size  for 
ACE/CID  training  would  have  exceeded  Bloom’s  challenge. 

7.  Comparisons  of  ACE/CID  Students  and  Fleet  ITs 

Table  5  shows  data  comparing  the  background  of  ACE/CID  students  with  that  of  the 
Elect  ITs  who  participated  in  IWAR-E.  All  Elect  ITs  were  rated  IT2  or  ITl.  One  student  had  only 
3  years  of  IT  experience  but  had  graduated  at  the  top  of  his  “A”  school  class.  Median  years  of 
Elect  IT  experience  was  8.  The  Knowledge  Test  identified  one  of  the  Elect  ITs  who  had  spent  all 
his  time  in  communications  and  had  virtually  no  experience  in  network  technology.  He  was 
replaced  by  another  (from  the  same  ship)  who  had  7.5  years  of  Elect  network  technology 
experience. 

Given  these  data,  it  seems  reasonable  to  conclude  that  the  Elect  ITs  detailed  for  IWAR-E 
were  representative. 
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Table  5.  Comparison  of  Fleet  ITs  and  ACE/CID  Students 


Comparisons 

Fieet  iTs 

ACE/CiD  Students 

Average  Age 

27.5 

20.8 

Average  Years  Navy  Experience 

8.5  (range  3-18) 

<  1 

Average  Years  IT  Experience 

7.2  (range  3-12) 

<  1 

Average  AFQT  Scores 

69.9 

69.0 

Total  “C”  Schools  Competed 

41 

0" 

Total  Navy  Certifications 

9 

0 

Male/Female 

9M/3F 

8M/4F 

Note  for  Table  5:  Some  “C”  school  material  was  included  in  ACE/CID  training. 

8,  IWAR-L  Results 

IWAR-L  was  divided  into  the  four  major  aetivities  described  earlier; 

1 .  Knowledge  testing  (4  hours  -  Day  1) 

2.  Troubleshooting  (13.25  hours  -  Days  1-3) 

3.  Security  testing  (3.5  hours  -  Day  4) 

4.  Design  and  implementation  (7  hours  -  Day  5). 

IWAR-L  results  are  reported  in  accord  with  these  four  activities. 

а.  Knowledge  Testing  (Day  1) 

The  written  Knowledge  Test  consisted  almost  entirely  of  short- answer  questions  on  the 
following  12  topics: 

1 .  Hardware AVindows  Basics 

2.  Client  Support  Fundamentals 

3.  Windows  Server  Fundamentals 

4.  Windows  Domain  Name  System  (DNS)  Servers 

5.  Active  Directory 

б.  Exchange 

7.  Group  Policy 

8.  CISCO  Internetwork  Operating  System  (lOS) 

9.  Open  Shortest  Path  First  (OSPF)  Protocol 

10.  Switching 

1 1 .  UNIX  Operating  System 

12.  Network  Security. 
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Knowledge  Test  items  were  “vetted”  for  relevanee,  eorreetness,  and  fairness  by  four 
members  of  the  Institute  for  Defense  Analyses  (IDA)  teehnieal  staff,  two  CID  instructors,  and  an 
FSET. 


Knowledge  Test  Procedures 

After  a  brief  introduction  to  the  I  WAR  exercises  and  the  Knowledge  Test,  both  groups  of 
12  Fleet  ITs  andl2  ACE/CID  students  took  the  test  together  during  the  morning  of  Day  1.  The 
test  was  administered  as  a  “closed  book”  test  in  2  parts  of  2  hours  each,  separated  by  a  half  hour 
break.  A  White  Team  member  was  present  at  all  times  to  answer  questions  and  to  act  as  a  test 
proctor. 

Knowledge  Test  Results 

Table  6  shows  the  performance  of  the  two  groups  on  each  Knowledge  Test  topic  and  then 
overall.  The  questions  were  scored  0-2,  with  0  for  an  incorrect  response  and  2  for  a  correct 
response.  Partial  credit  of  1  was  awarded  in  accord  with  an  anchored  scoring  rubric.  As  can  be 
seen  from  the  scores,  the  Knowledge  Test  was  challenging  for  all  participants. 


Table  6.  Knowledge  Test  Results 


Fleet  ITs 

ACE/CID  Students 

Topic 

No.  of 
Items 

AVG 

Std  Dev 

AVG 

Std  Dev 

Probability 

Effect  Size 

Hardware/Windows  Basics 

10 

11.7 

5.1 

14.7 

2.7 

p  <  .05 

0.74 

Client  Support  Fundamentals 

15 

12.5 

7.4 

21.8 

4.3 

p  <  .01 

1.54 

Windows  Server  Fundamentals 

17 

11.8 

5.4 

17.9 

3.8 

p  <  .01 

1.30 

Windows  DNS  Servers 

12 

3.6 

3.7 

11.9 

3.5 

p  <  .01 

2.31 

Active  Directory 

10 

2.0 

3.1 

6.3 

3.0 

p  <  .01 

1.42 

Exchange 

10 

8.7 

3.0 

10.9 

4.1 

ns 

0.63 

Group  Policy 

12 

3.8 

2.2 

6.1 

1.7 

p  <  .05 

0.87 

CISCO  lOS 

11 

4.4 

4.9 

15.4 

5.1 

p  <  .01 

2.11 

OSPF  Protocol 

11 

5.4 

4.8 

13.9 

2.3 

p  <  .01 

2.16 

Switching 

12 

8.9 

6.1 

16.4 

3.8 

p  <  .01 

1.27 

UNIX  Operating  System 

9 

4.9 

3.3 

1.6 

2.1 

p  <  .01 

-1.12 

Network  Security 

10 

9.0 

3.1 

9.8 

3.3 

ns 

0.26 

Total  Scores 

139 

86.7 

43.9 

146.7 

68.0 

p  <  .01 

1.02 

Note  for  Table  6:  ns  =  not  significant  at  p  <  .05. 


As  shown  in  Table  6,  the  ACE/CID  students  scored  significantly  higher  on  9  of  the 
12  test  topics  and  significantly  higher  overall,  with  a  total  score  of  146.7  compared  to  the  Fleet 
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total  score  of  86.7  and  an  effect  size  of  1.02.  Notably,  however,  their  scores  for  Security  were 
about  the  same  as  those  for  Fleet  ITs,  and  the  Fleet  ITs  outscored  the  ACE/CID  students  in  the 
UNIX  sub  test.  About  half  of  the  ACE/CID  students  missed  substantial  portions  of  the  UNIX 
training,  and  absence  of  that  training  shows  in  their  scores.  It  was  the  only  topic  for  which  any 
ACE/CID  students  received  a  score  of  0,  as  6  of  them  did. 

Scores  for  individual  Elect  ITs  were  spotty.  Individual  ITs  did  well  in  some  topics  but 
answered  almost  no  questions  in  others.  The  ACE/CID  students  were  more  uniform  in  their 
scores,  with  very  few  0  for  any  topics,  in  contrast  to  individual  Elect  ITs  who  either  seemed  to 
know  a  topic  well  or  not  at  all. 

If  the  Knowledge  Test  was  used  to  determine  whether  the  ACE/CID  students  were  at 
least  equal  in  IT  knowledge  to  experienced  Elect  ITs,  it  seems  reasonable  to  conclude,  conserva¬ 
tively,  that  they  are.  Overall,  they  outscored  the  Elect  ITs  by  about  68%,  for  an  effect  size  of 
1.02. 

Whether  the  same  content  and  methodology  can  be  implemented  successfully  in  the  DT 
remains  to  be  determined  in  Phase  2  of  the  DARPA  program.  However,  a  portion  of  the  DT  was 
available  and  used  for  ACE/CID  training.  The  IT  content  presented  by  the  DT  can  be  matched  to 
21  items  on  the  Knowledge  Test  (again  scored  0-2).  As  shown  in  Table  7,  ACE/CID  students 
produced  Knowledge  Test  scores  for  these  21  items  that  were  significant  both  statistically  and 
practically,  with  an  effect  size  of  1.73.  These  results  suggest  that  the  digitization  of  ACE/CID 
training  is  feasible  and  that  results  from  Phase  2  IWAR  may  be  similar  to  those  observed  in 
Phase  1. 


Table  7.  Knowledge  Test  Items  on  DT  Content:  Fleet  ITs  and  ACE/CID  Students 


Fleet  ITs 

ACE/CID  Students 

Number  of  Items 

AVG 

Std  Dev 

AVG 

Std  Dev 

Probability 

Effect  Size 

21 

18.0 

8.2 

28.75 

3.2 

p  <  .01 

1.73 

Note  for  Table  7:  12  ITs  and  12  ACE/CID  students. 

b.  Troubleshooting  (Days  1-3) 

Troubleshooting  exercises  took  up  about  half  of  IWAR-L  time.  These  exercises  were 
expected  to  provide  the  most  realistic,  valid,  and  representative  assessment  of  the  IT  capabilities 
needed  in  the  Elect.  The  laboratory  was  the  most  appropriate  setting  for  these  exercises  because 
it  allowed  the  introduction  of  catastrophic  and  intractable  problems,  such  as  losing  an  Exchange 
database,  cross-connected  switches,  looped  hubs,  viruses,  and  corrupted  Active  Directory.  These 
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problems  occur  in  the  Fleet,  but  their  introduction  for  experimental  testing  would  be  neither 
practical  nor  permitted  in  a  shipboard  environment. 

Most  IWAR-L  troubleshooting  items  were  drawn  from  an  analysis  of  1,093  Casualty 
Report  (CASREP)  trouble  tickets  in  Fleet  Remedy  databases.  The  problems  were  carefully 
developed  and  tested  by  the  technical  team;  vetted  for  correctness,  fairness,  representativeness, 
and  relevance  by  IDA,  CID,  and  FSET  technical  specialists;  categorized  for  problem  difficulty 
and  impact  on  ship  operations;  summarized  in  a  standard  format;  and  presented  to  IWAR-E  par¬ 
ticipants  in  the  form  of  trouble  tickets. 

Difficulty  of  troubleshooting  items  was  rated  on  a  5 -point  scale  ranging  from  very  easy 
(routine  problems  that  could  be  solved  by  the  average  “Power  User”)  to  very  difficult  (problems 
that  could  only  be  solved  by  the  most  proficient  and  experienced  IT  professionals).  Impact  was 
rated  on  a  lO-point  scale  keyed  to  the  number  of  IT  systems  (scope)  affected  by  the  problem  and 
the  impact  of  the  problem  on  ship’s  operations  (severity).  Impact  ranged  from  minimal  (one  sys¬ 
tem  affected,  one  specific  operation  impeded  but  not  halted)  to  catastrophic  (all  systems  affected, 
overall  functionality  severely  compromised).  Time  was  also  included  to  resolve  the  problems.  It 
was  based  on  subject  matter  expert  (SME)  estimates  of  the  time  needed  to  solve  a  problem  of 
given  difficulty  by  an  average  IT  capable  of  solving  it  (e.g.,  the  time  needed  by  a  very  proficient 
IT  to  solve  a  very  difficult  problem). 

Figure  1  shows  an  example  Troubleshooting  Summary  used  by  the  Technical  Team  to 
organize  and,  using  the  injection  script,  set  up  problems.  Trouble  tickets  were  given  to  IWAR 
participants  as  statements  of  the  problem  to  be  solved.  Figure  2  shows  the  trouble  ticket  asso¬ 
ciated  with  the  problem  summarized  in  Figure  1.  ITs  were  instructed  to  complete  the  trouble 
ticket  form  and  include  a  description  of  the  problem  solution  and  the  steps  taken  to  solve  it. 

Troubleshooting  System  Familiarization 

Each  group  of  ITs  was  given  an  hour  to  become  familiar  with  the  laboratory  systems. 
Instructions  were  read  from  a  script  by  members  of  the  White  Team,  who  were  assisted  by  mem¬ 
bers  of  the  Technical  Team  in  conducting  the  familiarization.  Familiarization,  along  with  all 
other  IWAR-E  activities,  was  recorded  by  video. 

Troubleshooting  Procedures 

Troubleshooting  sessions  were  initiated  with  a  problem  expressed  as  a  Trouble  Ticket 
given  to  each  of  the  two  ACE/CID  and  Fleet  teams.  The  same  sequence  of  troubleshooting 
problems  was  presented  to  the  ACE/CID  students  and  to  the  Fleet  ITs.  When  the  team  members 
finished  solving  a  problem  to  their  satisfaction,  or  they  gave  up,  they  requested  a  new  one. 
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Scenario: 

TS-SV-GC-30 

Concept  Tested: 

IP  configuration  and  Troubleshooting 

General  Description 


Add  a  static  route  for  the  172.16.0.0/30  network  to  point  to  172.16.1 .254 

Injection  Script 

1 .  Log  on  to  EX01  as  the  proctor  admin  account  (proctor) 

2.  Open  a  command  prompt 

3.  In  the  command  prompt,  enter  the  following  command: 
route  add  172.16.0.0  mask  255.255.252.0  172.16.1 .254 -p 

4.  To  test,  try  to  ping  WKS01 .  If  all  is  configured  correctly,  this  will  fail. 

Problem  Symptoms 

•  LT  Sulu  complains  he  is  not  receiving  email. 


Preferred  Solution(s) 


•  Delete  the  static  route  on  EX01 

Impact: 

If  clients  cannot  connect  to  EX01 ,  they  will  not  be  able  to 
send  or  receive  email. 

Impact  Rating: 

7-High 

Difficulty: 

Very  Hard 

Time  to  Resolve: 

30  minutes 

Figure  1.  Example  Troubleshooting  Problem  Summary 


Trouble  Ticket 


Day  3,  July  29,  Time  (Start/End) _ Team: 


Problem  Symptom:  Lt  Sulu  complains  he  is  not  receiving  email 


Problem  Solution: 


Key  Solution  Steps: 


Figure  2.  Example  Trouble  Ticket  as  Presented  to  ACE/CID  Students  and  Fleet  ITs 
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The  troubleshooting  problems  were  drawn  from  150  seenarios  prepared  for  IWAR-L.  As 
Figure  1  shows,  they  were  categorized  by  impact,  difficulty,  and  time  to  solve.  Although  differ¬ 
ent  scenarios  were  presented  in  the  two  IWAR-L  cycles,  sequencing  and  scenario  selection  were 
kept  the  same  for  the  proportion  of  concepts  tested,  the  levels  of  difficulty,  and  the  impact  for  the 
two  cycles.  Selection  and  sequencing  of  scenarios  was  done  by  IDA  and  was  known  only  to  IDA 
before  the  scenarios  were  presented. 

Outside  access  was  carefully  controlled  for  the  troubleshooting  sessions.  Cell  phones, 
email,  and  access  to  Google  were  not  allowed.  However,  an  attempt  was  made  to  mirror  ship¬ 
board  conditions  as  closely  as  possible.  Reference  materials  for  the  systems  were  available  in 
each  room,  and  the  participating  ITs  were  permitted  to  bring  and  consult  their  own  notes,  refer¬ 
ences,  and  troubleshooting  guides. 

Table  8  shows  the  percentages  of  troubleshooting  items  at  each  impact  and  difficulty 
rating  sequenced  for  Days  1-3. 


Table  8.  Percent  of  Troubleshooting  Problems 
at  Different  Levels  of  Impact  and  Difficulty  Presented  on  Each  Day 


Impact  Rating  Percent 

Difficulty  Rating  Percent 

Days 

1-2 

3-4 

5-6 

7-8 

9-10 

Very 

Easy 

Easy 

Medium 

Hard 

Very 

Hard 

1 

25 

25 

25 

25 

- 

30 

30 

40 

- 

- 

2 

20 

20 

20 

20 

20 

10 

20 

40 

30 

- 

3 

10 

25 

25 

20 

20 

- 

20 

30 

30 

20 

Troubleshooting  Results 

Table  9  shows  the  results  from  the  troubleshooting  exercises  for  the  four  ACE/CID 
teams,  the  four  Fleet  teams,  and  for  all  ACE/CID  and  Fleet  teams  combined.  These  data  suggest, 
again,  that  the  ACE/CID  students  did  well.  They  solved  about  25%  more  problems  than  the  Fleet 
ITs  solved. 

Each  problem  attempted  by  each  of  the  four  teams  (two  ACE/CID  and  two  Fleet  teams) 
was  scored  by  three  observers:  two  members  of  the  White  Team,  who  observed  troubleshooting 
directly  as  proctors,  and  one  member  of  the  Technical  Team,  who  observed  troubleshooting  indi¬ 
rectly  via  a  bank  of  multiple-image  monitors,  one  of  which  tracked  and  displayed  all  keyboard 
inputs  from  the  participating  ITs.  At  the  end  of  each  day  for  each  observed  problem,  the  three 
observers  discussed  all  the  ratings  they  had  assigned  and  the  reasons  for  assigning  these  ratings. 
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Table  9.  Troubleshooting  Test  Results 


Cycle 

Team 

No. 

Correctly 

Solved 

No.  of 
Correct 
Rated  5 
(Excellent) 

No.  Harmful 
Changes 
(3  to  5) 

Percent 

Problems 

Verified 

Percent 

Solutions 

Tested 

Fleet 

(1-1) 

19 

12 

4 

79% 

53% 

1 

Fleet 

(1-2) 

27 

22 

4 

81% 

81% 

ACE/CID 

(1-1) 

29 

27 

2 

100% 

97% 

ACE/CID 

(1-2) 

25 

20 

2 

88% 

92% 

Fleet 

(2-1) 

20 

12 

4 

90% 

85% 

2 

Fleet 

(2-2) 

13 

11 

6 

91% 

92% 

ACE/CID 

(2-1) 

23 

21 

2 

96% 

91% 

ACE/CID 

(2-2) 

22 

21 

2 

100% 

100% 

All  Fleet 
(95  attempted) 

79 

83% 

46 

18 

85% 

77% 

All  ACE/CID 
(102  attempted) 

99 

97% 

89 

8 

97% 

95% 

All  Fleet  vs. 

All  ACE/CID  significance 

p  <  0.01 

p  <  0.01 

p  <  0.01 

p  <  0.01 

p  <  0.01 

Note  for  Table  9:  The  numbers  in  the  “Team”  column  headings  indicate  the  team  and  the  cycle  (i.e., 
Fleet  (1-1)  =  Fleet  IT  Team  1/Cycle  1;  ACE/CID  (2-2)  =  ACE/CID  Team  2/Cycle2). 

Problem  solutions  fell  into  three  categories:  correct,  incorrect,  or  partially  correct.  Correct 
solutions  received  a  score  of  5,  and  incorrect  solutions  received  a  score  of  0.  Problems  in 
between  were  given  a  score  of  1-4,  depending  on  the  rater’s  judgment. 

The  three  observers  assigned  exactly  the  same  scores  to  about  75%  of  the  problems.  If, 
after  discussion,  the  observers  could  not  reach  agreement  on  a  rating,  the  reported  score  was  the 
average  of  the  three  separate  ratings.  Among  all  ratings  for  all  troubleshooting  problems,  only 
four  differed  by  more  than  one  point  on  any  scale. 

Changes  made  in  the  course  of  problem  solving  that  would  be  harmful  to  IT  operations 
and  not  returned  to  their  original  states  were  also  noted  and  rated  on  a  5-point  scale,  with  5  as 
most  harmful.  Many  trouble  tickets  found  in  Fleet  databases  resulted  from  such  changes  made 
deep  in  the  systems — changes  made  when  troubleshooting  other  problems  and  then  not  restored. 
ACE/CID  training  had  emphasized  the  importance  of  following  proper  procedures  in  trouble¬ 
shooting  (i.e.,  understanding  before  acting).  This  emphasis  is  evident  in  the  “Harmful  Changes” 
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ratings  contrast  between  Fleet  ITs,  who  had  a  total  of  18  harmful  changes,  and  ACE/CID  stu¬ 
dents,  who  had  a  total  of  8  harmful  changes. 

The  emphasis  on  proeess  is  also  evident  in  the  data  on  the  pereent  of  problems  verified 
and  the  pereent  of  solutions  tested.  ACE/CID  students  were  taught  to  verify  trouble  tieket  prob¬ 
lems  and  symptoms  carefully  before  beginning  to  solve  them.  They  were  also  taught  to  test  solu¬ 
tions  with  equal  care  to  ensure  that  they  had  genuinely  solved  the  problem  before  eoneluding  that 
they  had  done  so.  Again,  this  emphasis  on  problem-solving  proeedure  paid  off  in  the  ACE/CID 
scores  and  the  percent  of  problems  (97%)  and  solutions  (95%)  they  verified  vs.  the  pereent  of 
problems  (85%)  and  solutions  (77%)  the  Eleet  ITs  verified. 

Einally,  the  Eleet  ITs  attempted  95  total  troubleshooting  problems,  of  which  79  (83%) 
were  solved  correctly.  The  ACE/CID  students  attempted  102  troubleshooting  problems,  of  which 
99  (97%)  were  solved  correetly.  The  systematic  process  for  problem  solving  used  by  the  ACE/ 
CID  students  seems  evident  from  data  showing  that  almost  twiee  as  many  of  their  solutions  (89) 
were  rated  excellent  as  compared  to  those  of  the  Eleet  ITs  (46). 

All  the  differences  between  overall  Eleet  and  overall  ACE/CID  performance  were  signifi¬ 
cant  (p  <  O.OI). 

c.  Security  (Day  4) 

Realistic  Security  testing  required  the  use  of  the  single  physical  system  available  in  eaeh 
laboratory  room.  Eor  that  reason,  only  two  teams  eould  participate  in  the  Security  exercise  at  one 
time.  The  day  was  organized  so  that  the  two  Eleet  teams  worked  on  the  Security  problems  in  the 
morning  and  the  two  ACE/CID  teams  worked  on  the  problems  in  the  afternoon — 4  hours  for 
each  group. 

Security  Testing  Procedure 

The  Red  Team  preloaded  the  physical  systems  in  each  room  with  about  100  typical  ship¬ 
board  security  vulnerabilities  (e.g.,  insecure  system  aecounts,  open  ports,  missing  patches)  that 
had  been  identified  by  the  Spaee  and  Naval  Warfare  Systems  Center  (SPAWAR).  The  two  teams 
were  tasked  with  bringing  their  system  up  to  Eleet  speeification  while  avoiding  changes  that  did 
not  conform  to  Eleet  regulations. 

All  Eleet  ITs  had  unlimited  access  to  the  Internet.  Navy /Marine  Corps  Internet  (NMCI) 
files  would  be  downloaded  on  request  to  the  White  Team.  The  Information  Condition 
(INEOCON),  based  on  the  likelihood  of  cyber  attaeks,  was  initially  set  to  INEOCON  5  (Red, 
Critical).  After  120  minutes,  it  was  lowered  to  INEOCON  3  (Yellow,  Elevated).  Scoring 
procedures  were  determined  in  advance,  taking  into  aecount  the  severity  of  eaeh  vulnerability 
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and  whether  teams  made  corrections  that  were  not  authorized  by  the  appropriate  Program  of 
Record. 

Exercise  procedures  were  introduced  by  the  White  Team  in  accord  with  a  written  script. 
When  ACE/CID  or  Elect  IT  team  members  identified  a  vulnerability,  they  were  to  determine 
how  to  correct  it  and  then  inform  a  White  Team  member  of  their  findings  before  proceeding. 
Secondary  issues  that  would  take  excessive  time  to  execute  were  noted  but  then  waved  by  the 
White  Team.  At  the  end  of  the  testing,  the  Red  Team  assessed  the  team  results  by  checking  the 
system  and  the  White  Team  notes  to  determine  what  vulnerabilities  had  been  located  and 
whether  correct  actions  had  been  specified  or  performed  to  remove  them. 

Table  10  shows  the  points  scored  (out  of  a  possible  105)  by  each  of  the  four  teams,  along 
with  their  totals.  As  suggested  by  this  table,  the  Elect  IT  teams  achieved  higher  scores  on  this 
exercise  than  the  ACE/CID  teams.  This  difference  falls  just  short  of  statistical  significant  at 
p<  .05. 


Table  10.  Security  Testing  Results 


ACE/CID  Teams 

Fleet  Teams 

Topics 

Team 

1-1 

Team 

1-2 

Team 

2-1 

Team 

2-2 

Team 

1-1 

Team 

1-2 

Team 

2-1 

Team 

2-1 

Required  Patches 

4 

4 

10 

4 

0 

6 

8 

4 

Symantec  Mail  Security  for  Exchange 

0 

0 

0 

0 

0 

0 

0 

0 

Symantec  Server  Antivirus 

1 

1 

1 

1 

2 

7 

8 

8 

Service  Passwords 

0 

10 

4 

4 

4 

10 

5 

5 

Banner  Needs 

0 

0 

0 

0 

0 

0 

0 

0 

Unauthorized  Software 

0 

1 

0 

1 

4 

1 

4 

1 

Domain  User  with  Administrative  Rights 

0 

2 

0 

0 

0 

0 

0 

0 

INFOCON  3  Settings 

2 

12 

9 

6 

13 

18 

17 

14 

Switch  and  Router  Passwords  (Bonus) 

5 

3 

5 

5 

5 

0 

5 

0 

Team  Total  Points 

12 

33 

29 

21 

28 

42 

47 

32 

Overall  (420  possible) 

ACE/CID  Total  =  95  (23%) 

Fleet  Total  =  149  (35%) 

Note  for  Table  10:  The  numbers  under  the  word  “Team”  in  the  column  headings  indicate  the  team  and 
the  cycle  (i.e.,  Team  1-1  =  Team  1/Cycle  1). 


Security  testing  proved  to  be  unexpectedly  difficult  for  all  four  teams.  The  primary  issue 
appeared  to  be  the  inability  of  the  groups  to  execute  as  teams.  Teams  were  reminded  during 
Day  4  orientation  of  the  importance  of  teamwork.  They  were  told  to  organize  and  to  pick  their 
own  team  leader(s).  The  teams  picked  leaders,  but  they  failed  to  organize  as  a  unit.  Eor  instance, 
they  often  failed  to  identify  and  perform  tasks  that  could  be  done  in  parallel.  Some  secondary 
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failures  were  also  not  anticipated.  Some  teams  did  not  access  their  documentation,  or,  if  they  did, 
they  did  not  read  it  carefully.  Finally,  because  of  time  pressures,  the  one-on-one  tutorial  metho¬ 
dology  used  elsewhere  in  ACE/CID  training  was  abandoned  for  a  significant  portion  of  the  Secu¬ 
rity  training. 

d.  Design  and  Implementation  (Day  5) 

A  significant  proportion  of  Navy  ITs  are  deployed  in  non-Fleet,  non-shipboard  environ¬ 
ments  that  require  the  design  and  implementation  of  an  entire  IT  system.  Day  5  focused  on  these 
issues.  The  IWAR-L  participants  were  tested  on  their  ability  to  operate  in  an  environment  to 
which  they  had  never  been  exposed  and  to  understand  systems  with  which  they  had  never 
worked. 

Design  and  Implementation  Procedure 

Participating  ITs  were  given  a  rack  of  (physical)  equipment  and  a  set  of  objectives  that 
covered  matters  such  as  network  redundancy,  server  redundancy,  policy  requirements,  and  traffic 
optimization.  They  were  then  allowed  7  hours  to  design,  implement,  and  configure  a  new  Infor¬ 
mation  Technology  system  with  network  and  server  support  that  would  achieve  24  objectives, 
which  were  selected  for  IWAR-L  from  various  land-based  assignment  tasks. 

The  three  classes  of  objectives  were 

1.  Critical  (e.g.,  establish  a  Windows  domain;  enable  Internet  access  for  all  clients; 
automatically  configure  client  machine  Transmission  Control  Protocol/Internet  Pro¬ 
tocol  (TCP/IP)) 

2.  Secondary  (e.g.,  create/test  user  accounts  for  key  commanders;  display  a  warning 
banner  at  logon;  have  workstations  automatically  lock  after  5  minutes  of  inactivity) 

3.  Optimization  (e.g.,  distribute  server  roles  to  maximize  efficiency,  enable  DNS  serv¬ 
ers  to  resolve  IP  addresses  to  names). 

In  each  of  the  two  IWAR-L  cycles,  six  ACE/CID  students  worked  as  a  single  team  in  one 
laboratory  and  six  Fleet  ITs  worked  as  a  single  team  in  the  other  laboratory.  Each  group  chose  its 
own  leader.  For  this  exercise.  White  Team  observers  were  present  in  each  laboratory.  They 
scored,  with  comments,  the  design  and  implementation  produced.  Achievement  of  each  objective 
was  rated  on  a  6-point  scale,  with  0  representing  no  accomplishment  and  5  representing  full  and 
correct  achievement  of  the  objective.  Points  in  between  were  anchored  to  specific  implementa¬ 
tion  possibilities. 
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Design  and  Implementation  Results 


As  suggested  by  Table  11  (Day  5  objectives  successfully  achieved)  and  Table  12  (Day  5 
scores),  neither  group  did  well  in  this  exercise — successfully  achieving  only  about  60%  of  the 
objectives  and  receiving  a  little  over  half  of  the  total  possible  score  points.  At  the  end  of  Cycle  1, 
the  ACE/CID  team  had  successfully  achieved  three  more  objectives  than  the  Fleet  IT  team  (see 
Table  11)  and  was  slightly  ahead  in  the  scoring  (see  Table  12).  At  the  end  of  Cycle  2,  the  Fleet 
IT  team  had  successfully  achieved  four  more  objectives  than  the  ACE/CID  team  (see  Table  11) 
and  came  out  ahead  in  the  scoring  (see  Table  12).  This  difference  falls  just  short  of  statistical 
significance  at  p  <  .05. 


Table  11.  Design  and  Implementation  Objectives  Scored  4-5  (Successful) 


Cycle  1 

Cycle  2 

Objectives 

Fleet  IT 
Objectives 
Scored 
4-5 

ACE/CID 

Objectives 

Scored 

4-5 

Fleet  IT 
Objectives 
Scored 
4-5 

ACE/CID 

Objectives 

Scored 

4-5 

Fleet  IT 
Total 

Objectives 

Scored 

4-5 

ACE/CID 

Total 

Objectives 

Scored 

4-5 

Critical 
(7  objectives) 

5 

5 

2 

2 

7 

7 

Secondary 
(12  objectives) 

3 

6 

5 

1 

8 

7 

Optimization 
(3  objectives) 

0 

0 

0 

0 

0 

0 

Totals 

8 

11 

7 

3 

15 

14 

Table  12.  Design  and  Implementation  Scores  (Day  5) 


Cycle  1 

Cycle  2 

Objectives 

Fleet  IT 
Total  Scores 
(Out  of  110) 

ACE/CID 
Total  Scores 
(Out  of  110) 

Fleet  IT 
Total  Scores 
(Out  of  110) 

ACE/CID 
Total  Scores 
(Out  of  110) 

Fleet 

Total  Scores 
(Out  of  220) 

ACE/CID 
Total  Scores 
(Out  of  220) 

Critical 
(7  objectives) 

27.5 

24.7 

12 

11 

39.5 

35.7 

Secondary 
(12  objectives) 

31.5 

39.5 

35 

5 

66.5 

44.5 

Optimization 
(3  objectives) 

5.5 

3.3 

2 

1 

7.5 

4.3 

Totals 

64.5 

67.5 

49 

17 

113.5 

84.5 

Even  though  the  Cycle  2  Fleet  IT  team  scored  well  with  regard  to  the  specific  objectives 
and  scoring  rubric  established  for  Day  5,  White  Team  observers  pointed  out  that  the  team  had 
also  damaged  some  system  components  beyond  repair.  The  system  should  be  “slicked,”  and  the 
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team  should  either  start  over  or  reeeive  a  seore  of  zero  for  the  day.  The  scoring  rubric  will  have 
to  be  changed  for  Phase  2  IWAR  to  take  account  of  such  damage. 

As  in  the  Day  4  Security  exercise,  Day  5  tested  the  ability  of  the  groups  to  organize.  The 
groups  could  not  achieve  all  the  objectives  by  addressing  them  one  at  a  time.  The  groups  needed 
to  assign  different  members  to  work  on  different  objectives  that  could  be  accomplished  in  paral¬ 
lel.  Both  groups  in  both  cycles  had  difficulty  in  organizing  this  way.  The  systematic  approach 
taken  by  the  ACE/CID  students,  which  had  served  them  well  in  earlier  exercises,  slowed  their 
work  and  made  a  serial  approach  particularly  troublesome. 

During  the  off-site  lunch  period  of  Cycle  1 ,  some  members  the  Fleet  IT  group  attempted 
to  contact  outside  help.  Members  of  the  White  Team  successfully  interrupted  two  of  these 
attempts  before  any  advice  was  received.  A  brief  inquiry  did  not  reveal  any  successful  or  other 
attempts.  A  closely  supervised  lunch  was  provided  on-site  during  Cycle  2. 

F.  IWAR-D 

1.  IWAR-D  Organization 

IWAR-L  provided  a  comprehensive,  objective,  and  controlled  comparison  of  ACE/CID 
students’  knowledge  and  skills  with  those  of  experienced  Fleet  ITs.  IWAR-D  was  intended  to 
take  the  next  step  and  assess  the  dockside  performance  of  ACE/CID  students  using  installed 
shipboard  systems  (to  the  extent  permitted  by  shipboard  conditions  and  available  human 
observers). 

Initially,  IWAR-D  was  to  compare  the  performance  of  ACE/CID  students  with  Fleet  ITs, 
in  shipboard,  dockside  environments.  However,  given  the  results  from  IWAR-E  and  the 
constraints  of  shipboard  IT  activity,  such  a  comparison  appeared  to  deliver  little  additional 
information  relative  to  its  cost  and  effort.  As  subsequently  planned  with  Navy  coordination  and 
assistance,  IWAR-D  was  instead  designed  to  observe  only  the  performance  of  the  ACE/CID  stu¬ 
dents  in  shipboard,  dockside  environments. 

2,  IWAR-D  Schedule 

IWAR-D  consisted  of  two  5-day  cycles,  3-7  August  and  10-14  August  2009.  Cycle  1 
assessed  the  shipboard  performance  of  six  ACE/CID  students,  and  Cycle  2  assessed  the  perfor¬ 
mance  of  the  remaining  six  ACE/CID  students.  Two  ships,  USS  Preble  (DDG  88)  and  USS 
Milius  (DDG  69)  (see  Figure  3),  were  made  available  for  both  cycles.  Three  ACE/CID  students 
were  assigned  to  each  ship  in  each  cycle. 
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USS  Preble  (DDG  88) 


USS  Minus  (DDG  69) 


Figure  3.  Ships  Used  During  iWAR-D 

Two  primary  sources  of  data  were  obtained  from  IWAR-D;  a  two-page  survey  completed 
by  shipboard  supervisors  and  a  list  or  diary  completed  by  the  ACE/CID  students.  Figure  4  shows 
an  example  survey. 


(r)ri  INSTITUTE  FOR  DEFENSE  ANALYSES 

"  4*50MaACenterDm«/Alemdna,Virfnua22311-1882/Tefepbof* 703-845-2000 

Ttdmelagy  Diratm 

USS  Preble  (DDG-88)  (Cycle  2) 

Your  opinions  and  impressions  are  critically  important  at  this  stage  of  (WAR  testir>g. 
Please  compile  this  survey  at  the  end  of  each  day  this  week.  Thank  you  for  your  help! 

Date _ 

Preventive  Maintenance 

About  how  many  F»reven*w  Maintenance  tasks  did  the  DARPAflTs  work  on  today?  _ . 


HCTg diffigdmgfett;e Prewnt|»e^arwenanoejasks»si^gdjp DAR^/ms^ 


1  Vmv 

E»i 

Difficuli 

1  V/erv  DMcuR  1 

Whai  percent  d  Preventive  Mairrtertance  tasks  assigned  to  the  DARPA/TTs  (fed  they  complete 

succes^lv? 

1  0-20% 

20-40% 

1  40-60% 

eo-«o% 

1  80-100%  I 

What  were  Ihe  DARPA/ITs'  conihbutions  in  comotetino  Preventive  Maintenance  tasks  asskned  to  them? 

1  N(x>e 

Few/Minor 

1  Som^Averaoe 

Manv/Substartoal  1  Maior/Essential  1 

What  was  the  value  of  Preventive  MaaWenance  tasks  oomoleted  bv  toe  DARPA/ITs  to  shio  ooerations? 

1  Small 

Minor 

S<tostantial 

1_ Mss_ 1 

Whal  amouit  of  su 

eervision  was  reouired  lor  toe  DARPA/ITs'  Preventive  Mamenanoe  task  activities? 

1  C(X)stant  1  Freouent 

1  Occasional 

Rare 

1  None  1 

Are  there  comments  on  the  DARPA/TTs'  Preverrtiwe  Maintenance  performance,  value,  and  abilities  you'd 
like  to  ackl?  Please  write  them  here  andtor  on  the  back  of  this  sheet 


Scheduled  Maintenance 

About  how  many  Scheduled  Maintenarwe  tasks  (fed  the  DARPA/ITs  work  on  today? 


How diffio£ja«redie,Sd)eduled MaaitenanoejKk5,vsjgned to,DARP^ms^ 


1  Verv  Easy 

Bki 

Diffinult 

1  Verv  DAcult  1 

What  percent  of  Scheduled  Maintervance  tasks  assigned  to  toe  DARPA/ITs  was  compieled  successfJly? 

1  0-20% 

20-40% 

1  40-60% 

60-80% 

1  80-100%  1 

What  vrere  toe  DARPA/TTs'  contributions  to  complebng  the  Scheduled  Maintenance  tasks  assigned  to 

them? 

1  None 

Few/Minor 

1  Somtf Average 

Manw/Sitostanbal  1  Maior/Essential  1 

What  was  the  value  of  Schedided  Maeitenance  tasks  crvnoleted  bv  toe  DARPA/ITs  to  shto  ooerations? 

1  Small 

Minor 

Sttostanbal 

1_ Mator_ 1 

Whal  amoiait  of  su 

eervision  was  reouired  for  the  OARPA/ITs'  Scheduled  Mantenance  task  activities? 

1  Cortstant  1  Freouent 

1  OocasKvul 

Rwe 

1  None  1 

Am  there  comments  on  the  DARPA/TTs'  Scheduled  Matotenance  perfbrmartce.  value,  and  abiities  ycu'd 
Ike  to  add?  Please  write  them  here  and/or  on  the  back  of  ths  sheet 


Trouble  Ticket  Responses 

About  how  many  Trouble  Tickets  did  the  DARPA/ITs  work  on  today?  _ 


How  difficult  were  tt>e  Trouble  Ticket  t^s  assarted  to  DARPA/tTs? 


1  Verv  Fasv 

EffiV 

1  Averaoe 

1  DifFicidt  1 

Verv  Difficult  1 

What  peroentaoe  of  T rouble  Ticket  tasks  assarted  to  dte  DARPA/ITs  was  oomoleted  successfutiv? 

1  0-20% 

20-40% 

1  40-ao% 

1  60-80%  80-100%  1 

What  wore  the  DARPA/TTs'  contributions  to  comoletina  the  Trouble  Ticket  tasks  assioned  to  them? 

1  Nor>e 

Few/Mrur 

1  Some/Averaoe 

1  Manv/Substanbal  1 

Maior/Essentjal  1 

What  was  the  v^e  of  Trouble  Ticket  tasks  comoleted  bv  the  DAFV^A/ITs  to  shi 

D  ooerations? 

1  Small 

Minor 

1  Average 

1  Substantial 

1 

What  amoitot  of  su 

toervision  eras  reoiared  for  the  DARPA/ITs'  Trouble  Tek^  task  activities? 

1  Constant  1  Freouent 

1  Occasional 

Ess_ 1 

None  1 

Are  there  comments  on  the  DARPA/TTs'  TroouUe  Ticket  performance,  value,  arxl  abiities  you'd  Ike  to 
add?  Please  wrte  them  here  and/or  on  the  back  of  this  sheet 


Emerging  Problems 

About  how  many  Emergirtg  Problems  did  the  DARPA/ITs  work  on  today? 


How  difficult  were  tt>e  Emergirtg  ProblOT  tasks  assigned  to  DARPA/TTs? 


1  Veiy  Easy  I 

1  Easy 

1  Average 

1  Difficuk 

1  Verv  Dilicult  1 

What  oeroentaoe  of  EmercHr>o  Problem  tasks  assioned  to  the  OARPA/TT s  was 

>  completed  sucoessfufv? 

1  0-20%  1 

1  20-40% 

1  4040% 

1  60-80% 

1  80-t00%  1 

What  were  the  DARPA/ITs'  contribuborts  to  comoletina  the  Errterano  Problem  tasks  assioned  to  them? 

1  None  1 

1  Few/Minor 

1  Some/Averaoe 

1  Many/Substanbal  1  Maior/Essential  1 

What  was  the  value  of  Emeroira  Problem  tasks  completed  bv  the  DARPA/ITs  to  ship  operations? 

1  Smdl  1 

1  Minor 

1  Averaoe 

1  Sitostantial 

J_ Mator_ 1 

What  amount  of  su 

oervtsion  was  reouired  for  the  DARPA/ITs'  Emerair>g  Problem  task  activities? 

1  Constant  1  Freouent 

1  Occasional 

1  Rare 

1  Norte  1 

Are  there  comments  on  the  DARPA/TTs'  Emergng  Problem  performance,  value,  and  abilities  you'd  like  t 
add?  Please  write  them  here  ar>d/or  on  the  back  of  iNs  sheet 


other  Activity  (please  identify) 


Figure  4.  Example  IWAR-D  Survey 

The  survey  consisted  of  four  main  sections:  Preventive  Maintenance,  Scheduled  Main¬ 
tenance,  Trouble  Ticket  Responses,  and  Emerging  Problems.  The  Emerging  Problems  tasks  were 
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those  found  while  eorrecting  other  problems.  The  survey  included  a  final,  free  form  section 
called  Other  Activity,  which  could  be  used  to  discuss  tasks  that  did  not  fit  into  the  other  four 
categories. 

The  diaries  consisted  of  a  straightforward  list  of  tasks  attempted  and  the  actions  taken  to 
accomplish  them.  In  general,  the  diaries  appear  to  provide  a  more  accurate  description  of  the 
IWAR-D  activities  and  results  than  the  surveys  provided.  For  instance,  one  ship’s  set  of  survey 
responses  reports  58  tasks  for  Cycle  2,  with  exactly  the  same  number  of  tasks  in  each  of  the  four 
task  categories  in  3  of  the  4  days  reported.  These  survey  responses  do  not  match  the  ACE/CID 
student  diary  reports  for  that  week,  which  list  a  total  of  8  tasks  assigned  that  week  and  is  precise 
in  describing  each  task  and  the  steps  taken  to  accomplish  it.  Surveys  from  that  ship  for  its  other 
cycle  report  also  do  not  match  the  tasks  assigned  as  described  in  the  respective  task  diaries. 

Diary  and  survey  reports  from  the  other  ship  differ  much  less.  These  reports  and  the  data 
reported  here  are  based  on  survey  data  from  that  ship  and  the  task  diaries  from  both  ships. 

3,  IWAR-D  Results 

Overall,  the  ACE/CID  students  completed  about  38  tasks  during  IWAR-D — 16  in 
Cycle  1  and  22  in  Cycle  2.  The  number  of  tasks  completed  was  affected  by  shipboard  schedules 
and  activities.  Eor  example,  no  tasks  were  performed  for  2  days  aboard  the  Preble  in  Cycle  1 
because  of  prescheduled  drills  and  exercises.  The  categories  assigned  to  tasks  differed  consider¬ 
ably  between  the  ship  surveys  used  here  and  the  task  diaries,  so  all  tasks  are  reported  together. 
However,  it  does  appear  that  most  of  the  tasks  were  either  trouble  ticket  responses  or  emerging 
problem  tasks.  Very  few  preventative  maintenance  or  scheduled  maintenance  tasks  were 
assigned  to  the  ACE/CID  students.  Assigning  the  values  1-5  to  each  survey  dimension  (task  dif¬ 
ficulty,  tasks  successfully  completed,  student  contributions  to  completion,  value  of  task  comple¬ 
tion  to  ship’s  operations,  and  amount  of  supervision  required)  lead  to  the  results  shown  in 
Table  13. 


Table  13.  Survey  Reponses  Concerning  Tasks  Assigned  to  ACE/CID  Students 


Dimension 

Average 

Comment 

Task  difficulty 

3.6 

Above  average 

Tasks  successfully  completed 

4.3 

About  87%  successfully  completed 

Student  contributions  to  completions 

3.8 

Above  average 

Value  of  task  completion  to  ship’s  operations 

3.9 

Above  average 

Amount  of  supervision  required 

3.6 

Occasional  to  rare 
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The  task  diaries  and  the  surveys  do  not  indicate  whether  the  ACE/CID  students  were 
assigned  any  tasks  that  they  were  not  prepared  to  solve.  Some  tasks  were  not  solved  because  the 
ACE/CID  students  were  not  permitted  access  to  a  component  or  because  the  time  allocated  for 
the  day’s  IWAR-D  testing  ran  out. 

Most  of  the  ACE/CID  students’  contributions  to  their  tasks  were  rated  as  substantial  or 
essential.  In  two  instances,  the  ACE/CID  students  solved  problems  that  the  Elect  ITs  had  not 
been  able  to  solve  either  with  their  own  solutions  or  with  solutions  they  had  subsequently 
received  from  SPAWAR. 

The  ACE/CID  students  solved  about  half  of  their  tasks  with  little  or  no  supervision. 
Supervision  of  the  ACE/CID  students  typically  diminished  in  the  IWAR-D  cycles  after  the  ship¬ 
board  ITs  had  an  opportunity  to  observe  their  performance. 

G.  IWAR-A 

1,  IWAR-A  Organization 

IWAR-A  was  the  final  stage  of  Phase  1 .  It  was  intended  to  assess  the  performance  of 
ACE/CID  students  deployed  aboard  ships  at  sea.  As  with  IWAR-D,  testing  was  to  use  shipboard 
systems  and  employ  human  observers  to  the  extent  permitted  by  both  shipboard  and  deployment 
conditions. 

2.  IWAR-A  Schedule 

Again,  because  of  the  wealth  of  data  provided  by  IWAR-E,  likely  additional  information 
to  be  obtained,  and  constraints  of  time  and  resources,  IWAR-A  was  limited  to  observations  of 
three  ACE/CID  students  who  served  aboard  the  USS  Vandegrift  (EEG-48)  (see  Eigure  5)  on 
17-20  August  2009.  The  students  selected  for  IWAR-A  were  the  three  whose  next  duty  stations 
were  shore  based. 


-  f 


USS  Vandegrift  (FFG-48) 
Figure  5.  Ship  Used  During  iWAR-A 
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3.  IWAR-A  Results 


IWAR-A  data  consists  of  the  task  diaries  kept  by  the  ACE/CID  students  and  a  narrative 
report  from  the  Senior  Chief  who  served  as  their  monitor.  From  the  start,  the  ACE/CID  students 
were  assigned  system-wide  tasks  that  are  typieally  addressed  by  senior  ITs  who  have  taken  pre¬ 
requisite  “C”  sehool  eourses  and  have  several  years  of  fleet  experienee. 

On  the  first  day,  the  students  made  registry  ehanges  on  several  servers  to  install  anti-virus 
software  on  the  ship’s  Seeret  Internet  Protoeol  Router  Network  (SIPRNET),  reloaded  a  router 
that  was  not  routing  but  was  not  reporting  any  problems,  and  added  the  Exehange  address  into 
the  registry  key.  These  aetions  were  essential  for  sueeessful  operation  of  the  ship’s  Information 
Technology  systems.  If  any  were  done  ineorreetly,  the  results  would  have  been  debilitating. 

On  the  second  day,  the  ACE/CID  students  diseovered  a  potential  “emerging  problem” — 
one  of  baek-up  systems  was  not  funetional — that  had  not  been  reported.  The  problem  was  solved 
when  the  ACE/CID  students  installed  the  eorrect  software  on  the  system  and  informed  the  rest  of 
the  system  of  its  availability,  when  and  if  needed. 

The  work  of  the  ACE/CID  students  eontinued  in  the  third  and  fourth  day  as  regular  mem¬ 
bers  of  the  ship’s  IT  eompany. 

Despite  being  in  a  new  and  unfamiliar  environment,  the  three  ACE/CID  students  appear 
to  have  followed  the  eareful  and  systematie  proeesses  they  had  been  taught.  Evidenee  in  the  task 
diary  showed  that  they  (1)  cheeked  to  see  what  the  eore  problem  was  and  whether  it  matched  the 
symptoms  reported,  (2)  eorreeted  the  problem  without  making  aetual  or  potentially  harmful 
changes  in  the  system,  and  (3  )  ensured  that  their  work  eorreetly  solved  the  problem. 

All  three  ACE/CID  students  participated  equally  in  technieal  activities.  The  sueeess  of 
their  brief  tour  was  not  a  matter  of  a  single  superior  student  solving  problems  for  all  three. 

H.  Discussion 

The  IWAR-E  results  suggest  that  the  ACE/CID  training  aehieved  its  goal:  to  demonstrate 
that  the  eontent  and  methodologies  planned  for  the  computerized  DT  eould  aehieve  the  DARPA 
objeetive  of  produeing,  in  16  weeks,  new  sailors  who  have  the  knowledge  and  skills  of  experi- 
eneed  Fleet  ITs. 

In  the  Knowledge  Testing  and  Troubleshooting  exereises,  the  results  appear  to  have 
exeeeded  DARPA’s  goal  of  showing  parity  between  the  two  groups.  The  ACE/CID  students 
were  superior  in  these  aetivities.  The  Fleet  ITs  eame  out  ahead  in  the  Day  4  Security  scores, 
although  the  differenee  fell  short  of  statistieal  signifieanee.  This  result  may  refieet  the  faet  that 
the  one-on-one  instructional  methodology  used  elsewhere  in  ACE/CID  training  was  abandoned 


25 


for  portions  of  the  Security  training.  The  Fleet  ITs  also  scored  higher  in  the  Day  5  Design  and 
Implementation  exercises,  but  these  scores  again  fell  short  of  statistical  significance.  The  White 
Team  expressed  reservations  about  the  Fleet  IT  scores  because  of  damages  they  made  to  the 
systems  in  implementing  IWAR  objectives.  The  scoring  rubric  for  Day  5  will  need  to  be  changed 
for  Phase  2  IWAR. 

As  stated,  the  DARPA  Education  Dominance  Program  is  a  research  effort.  Much  is  being 
and  will  be  learned  as  it  progresses,  which  seems  appropriate  for  a  program  that  has  ambitious 
goals  of  considerable  consequence  to  the  Navy.  Testing  programs  as  elaborate  and  extensive  as 
IWAR-L  are  usually  preceded  by  careful  pilot  testing  of  materials  and  procedures.  Between  the 
pace  of  development  and  the  need  to  provide  data  that  would  reliably  inform  decisions  con¬ 
cerning  Phase  2,  neither  resources  nor,  especially,  time  was  available  for  pilot  testing.  The  orga¬ 
nizers  and  conductors  of  IWAR-L  are  to  be  congratulated  for  ensuring  that  it  operated  as 
smoothly  as  it  did. 

Overall,  the  ACE/CID  training  appears  to  have  more  than  achieved  its  objectives.  Using 
the  content  and  methodology  to  be  incorporated  into  the  DT,  ACE/CID  training  produced  stu¬ 
dents  who  demonstrated  equal — ^and,  in  many  cases,  superior — knowledge  and  skill  compared  to 
that  of  experienced  Elect  ITs. 

Phase  2  of  the  DARPA  Education  Dominance  Program  will  show  whether  ACE/CID 
content  and  methodology  can  be  digitized  for  the  DT  and  then  prove  to  be  as  successful  as  it  was 
in  IWAR  Phase  I .  Success  in  Phase  2  would  hold  considerable  promise  not  just  for  Navy  Infor¬ 
mation  Technology  training,  but  also  for  large  portions  of  Department  of  Defense  (DoD)  special¬ 
ized  skill  training.  The  superior  performance  of  ACE/CID  students  over  Elect  ITs  on  the 
21  DT-related  IWAR  Knowledge  Test  items  (see  Table  7)  suggests  that  the  DARPA  Educational 
Dominance  Program — ^if  successful  in  Phase  2 — should  advance  the  state  of  training  art  and 
practice  and  contribute  significantly  to  Navy  and  DoD  operational  capabilities. 
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